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INTRODUCTION 



Federal CSRD Legislation 

School improvement is increasingly viewed as an ongoing and comprehensive process. 
Recent legislation has encouraged the adoption of such a view: in 1 994, Congress altered regulations 
to allow those schools receiving Title I funds with the percentage of students receiving free and 
reduced lunch at 50% and above to use such funds for whole school improvement (American 
Institutes for Research, 1 999). Later, as part of the FY 98 Labor-HHS-Education Appropriations Act 
(PL 1 05-78), and again in 1999, Congress designated $ 1 50 million to promote comprehensive school 
reform. Much of these Comprehensive School Reform Demonstration (CSRD) funds flowed 
through states through a competitive grant process to schools and districts interested in implementing 
schoolwide, comprehensive reform. Of these monies, $120 million came from Elementary and 
Secondary Education Act (ESEA) Title I funds allocated by formula to states to allow current Title 
I schools to adopt comprehensive school reform models. The Fund for the Improvement of 
Education (FIE) allocated $25 million to flow to states by school-age population to allow non-Title 
I schools to implement reform programs. In addition, $4 million were allocated to Regional 
Educational Laboratories to assist states, districts, and schools in the implementation of the CSRD 
program. The remaining $1 million was used by the U.S. Department of Education to inform states 
and local schools about existing comprehensive reform models. 

According to the New American Schools (NAS) organization, which has developed several 
of the schoolwide reform models adopted by CSRD-fimded schools, “the success or failure of the 
CSRD Program has enormous implications for public education in the country’s poorest 
neighborhoods.” Title I of the ESEA of 1965, which provides federal funding for disadvantaged 
students, is due for congressional reauthorization this year. “To encourage research-based and 
replicable best practices,” Congress may reauthorize Title I based upon lessons learned from CSRD 
(NAS). 
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AEL CSRD Program 

AEL, Inc. began assisting Kentucky, Tennessee, Virginia, and West Virginia, the four states 
in its region, in early 1998 as part of the effort to support state departments of education in their 
implementation of CSRD. Assistance to state departments has included consultation and technical 
assistance concerning the development of procedures and criteria to select competitively the schools 
that would receive assistance under this program. Also, AEL has collaborated with state departments 
in developing procedures for guiding interested districts and schools through the process of assessing 
their needs, selecting among research-based reform models, or choosing to develop their own 
research-based comprehensive program. Also in collaboration with state departments, AEL has 
developed and provided workshops and technical assistance to schools and districts preparing their 
applications for grants under this program. Such workshops have included information about 
candidate reform programs, assistance with compiling aggregate components to create 
comprehensive programs for those not opting to purchase a model, and guidance regarding how to 
evaluate the effectiveness of programs implemented. 

Another significant component of AEL’s CSRD program is an Academy for External 
Facilitators. External facilitators are individuals assigned by state departments of education to assist 
schools implementing CSRD-fimded reform initiatives. AEL’ s Academy offers training and support 
as external facilitators execute five challenging roles in school reform: using the knowledge base on 
effective practices leading to improved student performance; modeling collaboration and joint 
problem solving; collecting, organizing, and analyzing school data to make decisions; brokering and 
recommending resources; and acting as agents of change in contexts in which facilitators possess 
no authority to direct change. 

Other AEL CSRD services have included a research symposium on CSRD and schoolwide 
reform and several independent and collaborative research efforts. 
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Formative Evaluation Process for School Improvement (FEPSI) 

Still another important facet of AEL’s CSRD program work has been to assist states and 
schools evaluate the effectiveness of programs implemented with CSRD funds. Participating CSRD 
schools are required to evaluate their progress toward nine CSRD implementation criteria 
(evaluation itself being one of the nine criteria). These nine criteria are 

(1) use of research-based methods/models 

(2) comprehensive design, in which the model(s) chosen are aligned to curriculum, instruction, and 
organization in the school 

(3) professional development 

(4) development of measurable goals and benchmarks 

(5) establishment of support within the school 

(6) parent and community involvement 

(7) consultation of external support and assistance 

(8) use of evaluation strategies 

(9) coordination of resources (Education for the Disadvantaged, 1998; see also AEL & CREP, 1 999) 

In May 1998, a partnership began between AEL, Inc. and the Center for Research in 
Educational Policy (CREP) at The University of Memphis to aid schools in their endeavor at self- 
evaluation. CREP and AEL designed a process called the Formative Evaluation Process for School 
Improvement (FEPSI) that incorporates several instruments and procedures developed by CREP. 
FEPSI is a package of services provided mainly during two visits per school year, one each in the 
fall and spring, by a trained individual called the site researcher. The fall and spring visits each 
consist of a different slate of data collection activities. 

During the fall visit, the site researcher conducts (1) a principal interview, (2) a School 
Observation Measure™ (SOM) comprised of several classroom visits, and (3) a benchmark-writing 
session with a committee of faculty. The principal interview is a 1-hour forum for learning a 
principal’s perceptions of CSRD implementation in his or her school. The SOM, an instrument 
developed by CREP, is a summary often 15-minute classroom observations typically conducted in 
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the course of a few hours (see both parts in Appendix A). In brief, the items on which classrooms 
are observed are best practices in instruction and instructional orientation that characterize the 
intended nature and organization of CSRD schools. Also adapted and designed by CREP, 
benchmarking is a process by which schools chart the course of CSRD implementation relative to 
the nine criteria; the resultant benchmark document is intended to be a signatory piece that a school 
owns and refers to over the course of three years of implementation and thereafter (see Appendix B 
for a page of a sample benchmark document). 

In the intervening period between the two visits, schools receive technical assistance and 
suggestions for revision on the first draft of their benchmark document by phone and mail from an 
AEL/CREP staff member. Also during this period, eight additional SOMs are conducted by a 
school’s external facilitator, which are added to the data collected by the site researcher. 

During the spring visit, the site researcher conducts (1) another SOM, (2) a focus group with 
a random selection of teachers, (3) a benchmark review with the committee that wrote the benchmark 
document in the fall, (4) a whole faculty meeting, and (5) an interview design process.' Similar to 
the principal interview, the focus group is a forum for learning teachers’ views of CSRD 
implementation. The benchmark review involves marking the school’s progress along a continuum 
expressed within the benchmark document. The purpose of the faculty meeting is to administer two 
questionnaires, the School Climate Inventory^ (SCI) and the Comprehensive School Reform Teacher 
Questionnaire^ (CSRTQ), both developed by the Center for Research in Educational Policy (AEL 
& CREP, 1999). The SCI assesses school climate along seven dimensions, consisting of 49 
statements set to a 5-point Likert-type scale. The CSRTQ assesses a school’s movement toward 
school reform along four dimensions — ^professional development, resources, pedagogical change, 
and outcomes — and also its specific progress toward school benchmark goals. Interview design is 
a musical chairs way of collecting data from many individuals in a small amount of time. All 
participants engage in asking and answering questions and analyzing responses. This procedure was 



'interview design is a technique used at the Wharton School, Management and Behavioral Science Center, 
introduced to AEL by Oralie McAfee (AEL & CREP, 1999). 
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incorporated into the FEPSI package as a way of gathering data from individuals from role groups 
other than the faculty; those to be included in the interview design process are community members, 
parents, and students. 

If it is possible to coordinate, it is desirable that a school’s external facilitator accompany the 
site researcher on both the fall and spring visits to a school. Following each visit, the site researcher 
is responsible for shipping data in appropriate formats to CREP. Based on the data collected by the 
site researcher and the external facilitator, the staff of the AEL/CREP partnership prepare and mail 
a report to each participating school at the end of the school year. Schools are assured of 
confidentiality and encouraged to use the report, along with their benchmarks, to adjust their 
activities to enhance CSRD implementation. 

Originally, FEPSI was introduced to the four states in AEL’s region: Kentucky, Tennessee, 
Virginia, and West Virginia. Kentucky, Tennessee, and West Virginia required their CSRD schools 
to use FEPSI, while Virginia encouraged its CSRD schools to use it. In total, 101 schools^ in these 
four states adopted FEPSI, purchased at a reasonable cost through the AEL/CREP joint initiative. 
Eighteen educators from these four states were trained as site researchers in September 1999, to 
facilitate the FEPSI process for schools. 

Georgia FEPSI Training 

After learning of FEPSI, CSRD staff from the Georgia Department of Education contacted 
AEL/CREP staff to express an interest in receiving FEPSI training and services for educators and 
CSRD schools in Georgia. The introduction of FEPSI to Georgia educators also served the purpose 
of being an initial step in marketing FEPSI beyond the four states in AEL’s region and CREP’s 
original area of practice (Tennessee). 



^ Ninety-six of the 101 schools were CSRD schools and participated in FEPSI to fulfill the evaluation 
component of the nine criteria. Five were not CSRD schools; however, they were implementing a research-based model 
and so chose to use FEPSI to chart their progress in model implementation. 
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The training was held October 11-13,1 999, at the Holiday Inn Conference Center in Macon, 
Georgia. Its major components were an overview of CSRD and PEPSI, instruction in how to 
conduct a SOM (which included practice SOMs in area schools), and instruction in benchmarking. 
A supplementary handbook compiled in a three-ring binder was given to each participant at the 
commencement of the training. It included overviews of CSRD and PEPSI, the scope of work for 
site researchers, sections explaining the various PEPSI activities, and some of the protocols to be 
used. An additional resource was provided during the instruction on SOMs, a booklet of definitions 
for the practices to be observed during a SOM (CREP, 1 999). 

Purpose and Objectives of Evaluation 

In an ongoing effort to improve its services and to satisfy its obligations to the U.S. 
Department of Education’s Office of Educational Research and Improvement, AEL evaluates all of 
its activities associated with regional educational laboratory work. Accordingly, evaluation of AEL’s 
CSRD work is undertaken to assess these services and their potential outcomes and impact. This 
report is intended to inform staff as they continue to make decisions regarding the content and 
structure of training sessions in PEPSI. 

The objectives of this evaluation report are to 

• assess the quality of presentation of the PEPSI training in Georgia 

• assess the knowledge and skills gained by training participants 

• assess aspects of the training for future PEPSI trainings outside AEL’s region and CREP’s 
usual area of practice 

Audience 

The primary audiences for this report are CSRD staff at AEL and staff of the AEL/CREP 
partnership. Secondary audiences include other AEL staff and management, and researchers 
interested in formative evaluation of comprehensive school reform. 
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METHOD 

A survey evaluation was conducted of the PEPSI training in Georgia. 

Participants 

Thirty educators were listed as participants to be trained as site researchers to facilitate the 
PEPSI process in 75 Georgia schools.^ At any given time during the training, between 25 and 30 
participants attended. Twenty-two participants returned a completed evaluation form. 

Instrument 

An evaluation survey form comprised of forced-choice and open-ended response items was 
designed by CSRD staff and evaluators (see Appendix C). It presents 37 questions divided into five 
sections. Section A consists of general forced-choice questions about the quality of training 
presentation, set to a 5-point Likert-type scale. Section B consists of forced-choice questions about 
the training’s value in teaching specific knowledge and skills related to PEPSI; each of the items was 
set to two 5-point Likert-type scales, one measuring prior knowledge and one measuring knowledge 
gain. The remaining three sections consist of open-ended response questions about the utility of the 
training and materials relative to three distinct aspects of the training — overview of CSRD and 
PEPSI, how to conduct a SOM, and how to do and facilitate benchmarking. 

A Cronbach alpha, r - .94, was generated for Section A of the instrument comprised of 14 
items (based on 21 respondents). Thus, this part of the instrument, which gauged the quality of the 
training presentation, possessed high internal reliability. An alpha coefficient was not computed for 
the second section of questions, related to the attainment of knowledge and skills specific to PEPSI 
activities, because these items were independent of each other. 



^The 75 CSRD schools in Georgia were required to use FEPSI services. Site researchers were each assigned 
to serve one to several schools. 
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Procedure 

The evaluation form was distributed to participants at the conclusion of the 3-day training. 
Evaluation staff were not present at the training; therefore, the form was distributed by AEL CSRD 
staff who had attended the training. Completed forms were collected by one of Georgia’s CSRD 
staff (and also a training participant) and mailed to AEL CSRD staff. 

Data Analyses 

Data were entered into SPSS 8.0 for Windows. Descriptive statistics on the forced-choice 
items of Sections A and B were generated. Open-ended response items in the last three sections 
were summarized. Analyses were conducted by members of the CSRD evaluation team. 

FINDINGS 

Twenty-two of the original 30 participants completed the evaluation form for this training. 
Section A 

For the first 1 4 questions related to the quality of training presentation, respondents answered 
on a 5-point Likert-type scale, anchor point 1 meaning not at all true and anchor point 5 meaning 
very much true. 

Responses to all 14 items had a mean of 4.1 or higher (see Table 1). The two items that 
garnered the highest ratings (as well as the least amounts of variation) were items 6 and 7: fVas 
conducted in a professional manner (4.7, SD .48) cind Was conducted by competent presenter(s) (4.8, 
SD .43). The lowest rated item was Had activities that were well sequenced (4.1, SD .71). Two 
other items with a comparatively lower rating also had the highest amounts of variation: Stimulated 
me to want to use the concepts . . . presented (4.2, SD .83) and Increased communication and 
collegiality with others (4.2, SD .80). 
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Table 1 



Descriptive Statistics for Section A of the Georgia PEPSI Training Evaluation Form 



Item 


N 


Mean 


SD 


1 . Had clear outcomes 


22 


4.5 


.67 


2. Included clear directions for activities 


22 


4.5 


.60 


3. Facilitated development of new skills 


22 


4.3 


.77 


4. Was conducted in an appealing manner 


22 


4.3 


.72 


5. Was conducted in a professional manner 


22 


4.7 


.48 


6. Was conducted by competent presenter(s) 


22 


4.8 


.43 


7. Had activities that were well sequenced 


22 


4.1 


.71 


8. Had activities that reinforced content 


22 


4.4 


.67 


9. Included appropriate examples 


22 


4.5 


.60 


10. Was relevant to my needs 


21 


4.6 


.51 


1 1 . Had meaningful involvement of participants 


22 


4.4 


.66 


12. Stimulated me to want to use the concepts, skills, and/or 
materials presented 


22 


4.2 


.83 


13. Increased communication and collegiality with others 


22 


4.2 


.80 


14. Increased my understanding of formative evaluation 


22 


4.6 


.60 



Section B 

For the next 1 1 questions about the training’s value in transmitting specific knowledge and skills 
relevant to FEPSI, respondents rated both their prior knowledge and knowledge gain. Respondents 
rated their knowledge gain most highly regarding Understanding the role of formative evaluation 
in the CSRD program (4.6, SD .75). They rated their knowledge gain the least regarding Technology 
skills (1.5, SD .87). See Table 2 for details. 
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The difference scores between prior knowledge and knowledge gain showed that on most items, 
respondents rated their knowledge gain less highly than their prior knowledge (see Table 2). This 
may indicate that participants already possessed a good deal of knowledge regarding these items. 
Two items that depart from this pattern are Understanding the role of formative evaluation in the 
CSRD program (2.1, SD 1 .27 for prior knowledge and 4.6, SD .75 for knowledge gain: difference 
score between means being 2.5) and Knowledge about Comprehensive School Reform models (2.8, 
SD .92 for prior knowledge and 2.8, SD 1 .23 for knowledge gain: difference score between means 
being 0). Note that the former item was also the most highly rated item for knowledge gain. 

Table 2 



Descriptive Statistics for Section B of the Georgia PEPSI Training Evaluation Form 





Prior knowledge 


Knowledge gain 


Diff. 


Item 


N 


Mean 


SD 


N 


Mean 


SD 


Score 


15. Understanding the role of formative 
evaluation in the CSRD program 


22 


2.1 


1.27 


20 


4.6 


.75 


2.5 


16. Knowledge of schools and 
classrooms 


22 


4.6 


.60 


21 


2.9 


1.26 


-1.7 


17. Learning data collection techniques 


22 


4.1 


.84 


22 


3.8 


1.18 


-0.3 


1 8. Oral and written communication 
skills 


22 


4.5 


.60 


22 


2.8 


1.33 


-1.7 


19. Facilitation skills 


22 


4.6 


.67 


21 


3.0 


1.24 


-1.6 


20. Skills in interpreting and analyzing 
qualitative and quantitative data 


22 


3.6 


.79 


22 


3.4 


.90 


-0.2 


21. Knowledge about and 

understanding of Comprehensive 
School Reform . . . 


22 


3.1 


1.23 


22 


3.4 


1.09 


0.3 


22. Knowledge about Comprehensive 
School Reform models 


22 


2.8 


.92 


22 


2.8 


1.23 


0.0 


23. Familiarity with common strategies 
associated with school reform . . . 


22 


4.1 


.84 


22 


3.0 


1.20 


-1.1 


24. Classroom observation skills . . . 


22 


4.3 


.77 


22 


3.4 


1.18 


-0.9 


25. Technology skills . . . 


20 


3.9 


.97 


21 


1.5 


.87 


-2.4 



er|c 



16 



11 



Section C 

The third section of the instrument posed these four questions related to the component of the 
training, Overviews on formative evaluation and Comprehensive School Reform'. (1) the most 
effective part of the presentation, (2) the quality of alignment between materials and presentation, 
(3) suggestions for improving this part of the presentation, and (4) efficacy of this training segment 
in preparing the participant for implementing PEPSI. 

Eighteen of 22 respondents answered the first question. What was the most effective part of the 
Overview presentations?, 3 giving more than one response. Six respondents said the presentation 
or presenter, referring to her or his knowledge, skills, expertise, or wit (some identified a presenter 
who did not present during this segment). Four additional respondents referred to a specific 
presentation technique that they appreciated: the use of analogies, the holistic picture, and the 
promise that overheads would be reviewed later. Four stated that the general CSRD or formative 
evaluation information or imparting of skills was the most effective aspect, 1 specifically saying the 
establishment of expectations for practice. Five respondents listed another component of the training 
instead of saying what was most effective about the overviews component: 4 referred to the SOM 
presentation and 2 referred to the benchmarking presentation. One respondent answered that she or 
he did not know. 

Eighteen of 22 respondents answered the second question. How well did the materials in your 
notebook align with the Overview presentations? Fourteen respondents answered affirmatively with 
a qualifying term such as excellent, great, very well, or good, or just a statement; 1 respondent, 
however, added that a lack of sequence existed and 1 added that the page numbering could have been 
better. Two respondents also answered affirmatively, but less strongly, indicating that they thought 
alignment was fair; 1 of these respondents also commented about a lack of sequence. Two final 
respondents indicated dissatisfaction, stating problems in alignment, “ ‘hopping’ around,” and lack 
of matching to sequence of overheads. 
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Eighteen of 22 respondents answered the third question, What suggestions would you have for 
improving the Overview presentations? Seven said none. Three suggested using an electronic 
medium for presentation, such as PowerPoint. Two suggested clarifying the time line of 
expectations. Finally, 6 respondents each made one of the following suggestions: make overheads 
available to trainees, rewrite the overheads so that each captures more concepts, number the pages 
of the manual consecutively, allow more time, allow more time for benchmarking, and revise 
presenter style so that it is not condescending. At least 2 of this latter group of responses seem to 
have been referring to another presentation. 

Nineteen of 22 respondents answered the fourth question. How well did the Overview 
presentations prepare you for implementing the Formative Evaluation Process for School 
Improvement? Thirteen answered affirmatively with a qualifying term such as very well ox good or 
with a statement that they felt prepared. Another 3 respondents also answered affirmatively, but less 
strongly, with a qualifying term such as fair or average. Still another respondent said OK. One 
respondent stated a need for the overheads. A final respondent said that she or he did not know. 

Section D 

The fourth section of the instrument posed these four questions related to the component of the 
training on Benchmarking: ( 1 ) the most effective part of the presentation, (2) the quality of alignment 
between materials and presentation, (3) suggestions for improving this part of the presentation, and 
(4) efficacy of this training segment in preparing the participant for implementing FEPSI. 

Eighteen of 22 respondents answered the first question. What was the most effective part of the 
Benchmarking presentations?, 3 giving more than one response. Eight respondents listed the 
preview of benchmarking and/or the examples presented during the preview. Four said the 
opportunity to practice benchmarking, 1 adding that they needed more practice. Three said the 
directions or hints given about benchmarking. Two respondents said that the clarification of the type 
of benchmarks vmder study helped them. Two respondents listed the presenter, and 1 respondent 
listed the handbook. One response could not be coded. 
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Nineteen of 22 respondents answered the second question, How well did the maten'uts in your 
notebook align with the Benchmarking presentations? Fifteen answered affirmatively with a 
qualifying term such as very well or good or with a statement. Various comments by these 
respondents were that materials were easy to follow and helpful and that they appreciated the 
benchmarking booklet and the extra information. One other respondent also answered affirmatively, 
saying alignment was OK. Two indicated that alignment was not good, 1 saying that the materials 
did not match the sequence on the overheads. One respondent did not respond directly, but stated 
that she or he would have liked for the session to have progressed from the draft to the commercial 
stage. 

Seventeen of 22 respondents answered the third question. What suggestions would you have for 
improving the Benchmarking presentations? , 5 giving more than one response. Seven expressed a 
desire for more time to practice benchmarking, 1 advocating a slower pace and 2 stating the need for 
more time to receive feedback as well as write benchmarks. Three respondents recommended a 
different procedure for engaging in practicing benchmarks, 2 suggesting that participants have the 
chance to work alone as well as together and 1 suggesting that they learn how to progress from the 
draft to the professional stage. Four respondents expressed a desire for more examples, more 
information on the models, or a greater variety of models. Two respondents suggested scheduling 
the benchmarking presentation for a different time of day. One would have liked to have seen a time 
line for how benchmarking should be accomplished in schools. One needed help distinguishing the 
responsibilities of the site researcher versus those of the school in writing benchmarks. Four 
respondents had no suggestions. 

Eighteen of 22 respondents answered the fourth question. How well did the Benchmarking 
presentations prepare you for implementing the Formative Evaluation Process for School 
Improvement? Nine respondents answered affirmatively with a qualifying term such as super, very 
well, good, or competent, 1 adding that she or he would appreciate follow-up after facilitating 
benchmarking at a school. Three other respondents also answered affirmatively, but less strongly, 
saying that they felt fairly or somewhat well-prepared. Six respondents said that they did not feel 
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confident, sure, or well-prepared to facilitate benchmarking, 3 adding that this was the one aspect 
of the training about which they felt this way. One respondent each from the latter two response 
groups qualified their feeling of unpreparedness by saying that experiencing the process of 
facilitating benchmarking in a school would help to acclimate them to it. 

Section E 

The fifth section of the instrument posed these four questions related to the component of the 
training on the School Observation Measure (SOM) : (1) the most effective part of the presentation, 
(2) the quality of alignment between materials and presentation, (3) suggestions for improving this 
part of the presentation, and (4) efficacy of this training segment in preparing the participant for 
implementing PEPSI. 

Twenty of 22 respondents answered the first question. What was the most effective part of the 
School Observation Measure (SOM) presentations?, 10 giving more than one response. Ten 
respondents said that the opportunity to practice in schools was the most effective part of the SOM 
segment, and correlated to that, 3 respondents said that debriefing after the practice was. Six 
respondents named the explanations and examples given in the presentation, and correlated to that, 

1 respondent listed the presenter. Five respondents listed the video. Two listed the handbook. 
Two respondents named the instrument itself, referring to its clarity or comprehensibility. One said 
all parts of the SOM training were the most effective. One respondent suggested that more vignettes 
and discussion were needed. Finally, another ancillary comment stated why the SOM segment of 
the training was effective compared to other parts of the training — because it allowed more time for 
lecture, practice, and clarification. 

Twenty-one of 22 respondents answered the second question. How well did the materials in your 
notebook align with the School Observation Measure (SOM) presentations? Twenty answered 
affirmatively with a qualifying term such as very well, well, great, excellent, or good, or with a 
statement. One respondent said OK. Several respondents made particular reference to a specific 
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training material: 4 mentioned the SOM notebook, 1 mentioned the training manual, and 1 
mentioned a “cheat sheet.” 

Fourteen of 22 respondents answered the third question. What suggestions would you have for 
improving the School Observation Measure (SOM) presentations? Six respondents said none. Four 
suggested the presentation of more examples for observation and/or more time for discussion, 1 
specifying that she or he would have preferred more video examples and time to debrief afterward. 
Three respondents each made one of the following suggestions for improvement: to have the training 
earlier in the year, to revise the response sheet, and to change the presenter’s style. One response 
could not be coded. 

Nineteen of 22 respondents answered the fourth question. How well did the School Observtion 
Measure (SOM) presentations prepare you for implementing the Formative Evaluation Process for 
School Improvement? Seventeen respondents answered affirmatively with a qualifying term such 
as very well, great, good, or fine, or with a statement using a term such as comfortable, prepared, 
competent, connected and logical. Another respondent also answered affirmatively, saying OK. One 
respondent added the suggestion that the training be held earlier in the year. One response could not 
be coded. 
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CONCLUSIONS 

Several conclusions may be drawn based upon the data. 

• Participants rated all items high (mean of 4. 1 or above on a 5-point scale) related to specific and 
general training objectives. 

• Participants clearly believed that the training was conducted by competent presenters, was 
conducted in a professional manner, and was relevant to their needs. 

• Participants appeared to learn most about the role of formative evaluation in the CSRD program. 

• Participants came to the training particularly knowledgeable of schools and classrooms and 
particularly skilled in written and oral communication as well as facilitation. 

• Participants had the least prior knowledge of comprehensive school reform and CSR models. 

• In general, participants seemed pleased with the training materials, although some specific 
suggestions for improvement were offered. 

• Participants found the benchmarking materials to be very helpful, but would have liked even 
more time to practice with the benchmarking process. 

• Participants felt least prepared to engage in the process of facilitating benchmarking in schools. 
This is likely due to the complexity of the task compared to others, rather than the quality of 
presentation itself 

• Participants were pleased with the hands-on practice with the SOM materials by video examples 
and visits to area schools and felt prepared to use the SOM materials. 
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RECOMMENDATIONS 

A few recommendations may be suggested, given the conclusions of this report. They may be 
consulted prior to planning other PEPSI trainings. 

• Consider the following finding in presenting future training: Some participants recommended 
that there be more logical flow in the training material, particularly as it corresponded to the 
flow of presentations (i.e., sequence to mimic overheads sequence, consecutive page 
numbering). 

• Continue to use hands-on activities and school visits for the SOM segment of PEPSI training. 

• Consider devoting more time and practice opportunities for the benchmarking segment of PEPSI 
training. Also, perhaps schedule it earlier in the day. 
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DIRECTIONS 



PLEASE DO NOT FOLD OR STAPLE THIS FORM. 



zmwi 

MAKI; UAKK MARKS 

i:kasi: compli:ti:ly i o changu School Observation Measure (SOM®) 

Data Summary 

For use in CSR formative evaluation conducted jointly by the 
Center for Research in Educational Policy, The University of Memphis and the 
AEL Regional Educational Laboratory, Charleston, WV. 



School Name Observer Name: 

Date of Observation: SOM# Observer Role/Affiliation: 

Number of classroom observations comprising this SOM. 

Directions: Use your class-specific notes to reflect upon the extent to which each of the 
following is present in the school: 

Instructional Orientation 

Direct instruction with the entire class (lecture) 

Team teaching 

Cooperative/collaborative learning 

Individual tutoring (teacher, peer, aide, adult volunteer) 

Classroom Organization 

Ability groups 
Multi-age grouping 

Work centers (for individuals or groups) 

Instructional Strategies 

Instructional feedback (written or verbal) to enhance student learning 
Integration of subject areas (interdisciplinary/thematic units) 

Project-based learning 
Use of higher-level questioning strategies 
Teacher acting as a coach/facilitator 
Parent/community involvement in learning activities 

Student Activities 

Independent seatwork (self-paced worksheets, individual assignments) 

Experiential, hands-on lesuming 

Systematic individual instruction (differential assignments geared to individual needs) 
Sustained writing/composition (self-selected or teacher-generated topics) 

Sustained reading 

Independent inquiry/research on the part of students 
Student discussion 
Technology Use 

Computer for instructional delivery (e.g. CAI, drill & practice) 

Technology as a learning tool or resource (e.g. Internet research, spreadsheet 
or database creation, multi-media, CD Rom, Laser disk) 

Assessment 

Performance assessment strategies 

Student self-assessment (portfolios, individual record books) 

Summary Items 

Academically focused class time 

Level of student attention/interest/engagement 
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(0) Not Observed: 
(1) Rarely: 

(2) Occasionally: 

(3) Frequently: 

(4) Extensively: 



Rubric for SOM Scoring 

Strategy was never observed. 

Observed in only one or two classes. Receives isolated use and/or little time in classes. 
Clearly not a prevalent/emphasized component of teaching and learning across classes. 

Observed in some classes. Receives minimal or modest time or emphasis in classes. 

Not a prevalent/emphasized component of teaching and learning across classes. 

Observed in many but not all classes. Receives substantive time or emphasis in classes. 
A prevalent component of teaching and learning across classes. 

Observed in most or all classes. Receives substantive time and/or emphasis in classes. 

A highly prevalent component of teaching and learning across classes. 
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Classroom Observation Notes for School Observation Measure (SOM) 

For use informative evaluation conducted jointly by the Center for Research in Educational Policy, The University of Memphis and 

AEL Regional Educational Laboratory, Charleston, WV 
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Student Activities 
Independent seatwork 
Experiential, hands-on learning 
Systematic individual instruction 
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APPENDIX B: 



One Page of a Sample Benchmark Document (AEL & CREP, 1999) 
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Formative Evaluation Process for School Improvement 
Orientation and Training 
Georgia Department of Education 
October 11-13, 1999 
Macon, Georgia 

Evaluation Form 



We are interested in continuously improving our servic^. Please help us to do so by taking a 
few moments to respond to the following items. Your replies will remain anonymous and 
confidential, so feel free to answer candidly. Thank you! 



Section A: Circle the number that best indicates the extent to which the orientation and training 
on the Formative Evaluation Process for School Improvement 



Not at 
all 



Very 

much 



1 . Had clear outcomes 

2. Included clear directions for activities 

3. Facilitated development of new skills 

4. Was conducted in an appealing manner 

5. Was conducted in a professional manner 

6. Was conducted by competent presenter(s) 

7. Had activities that were well sequenced 

8. Had activities that reinforced the content 

9. Included appropriate examples 

10. Was relevant to my needs 

1 1. Had meaningful involvement of participants 

12. Stimulated me to want to use the concepts, 
skills, and/or materials presented 

1 3 . Increased communication and collegiality 
with others 

14. Increased my understanding of formative 
evaluation 



1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 



2 3 4 5 
2 3 4 5 
2 3 4 5 
2 3 4 5 
2 3 4 5 
2 3 4 5 
2 3 4 5 
2 3 4 5 
2 3 4 5 
2 3 4 5 
2 3 4 5 



1 



2 



3 4 



5 



1 2 3 4 5 



1 2 3 4 5 
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Section Bt Listed below are items related to knowledge and skills to be obtained during the 
Orientation and Training sessions. Please rate each item on two dimensions: \) degree of 
knowledge/skill you had prior to training, and 2) degree to which knowledge/skill increased as a 
result of your participation. Scale ; 1 = None or not at all to 5 = Ver>' much or a lot 



15. 

16. 

17. 

18. 

19. 

20. 

21 . 



22 . 

23. 



24. 



25. 



Understanding the role of formative 
evaluation in the CSRD program 

Knowledge of schools and classrooms 

Learning data collection techniques 

Oral and written communication skills 

Facilitation skills 



Prior Knowledge 
1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



Skills in interpreting and analyzing 
qualitative and quantitative data 

Knowledge about and understanding of 
Comprehensive School Reform (e.g., 
relationship to Title I, school-wide 
programs, Obey-Porter legislation) 

Knowledge about Comprehensive 
School Reform models 



1 2 3 4 5 



1 2 3 4 5 

1 2 3 4 5 



Familiarity with common strategies 
associated with school reform (e.g., use 
of technology, cooperative and team- 
based strategies, student-centered learning) 



Classroom observation skills (e.g., 
recognizing and identifying what is 
happening in a class) 

Technology skills (e.g., e-mail capability, 
Internet access, data entry) 



Knowledge Gain 
1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



1 2 3 4 5 

1 2 3 4 5 



1 2 3 4 5 



1 2 3 4 5 

1 2 3 4 5 
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Section C: Thinking only of the Ovennews on Formative Evaluation and Comprehensive School 
Reform, please answer briefly the following questions. 

26. What was most effective part of the Overview presentations? 



How well did the materials in your notebook align with the Overview presentations? 



28. What suggestions would you have for improving the Overview presentations? 



29. How well did the Overvie^v presentations prepare you for implementing the Formative 
Evaluation Process for School Improvement? 



Section D: Thinking only of the Benchmarking sessions, please answer briefly the following 
questions. 

30. What was most effective part of the Benchmarking presentations? 



31. How well did the materials in your notebook align with the Benchmarking presentations? 



3 
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32. What suggestions would you have for improving the Benchmarking presentations? 



33. How well did the Benchmarking presentations prepare you for implementing the 
Formative Evaluation Process for School Improvement? 



Section E: Thinking only of the School Observation Measure (SOM), please answer briefly the 
following questions. 

34. What was most effective part of the School Obsei'vation Measure (SOM) presentations? 



35. How well did the materials in your notebook align with the School Observation Measure 
(SOM) presentations? 



36. What suggestions would you have for improving the School Observation Measure (SOM) 
presentations? 



37 . 



How well did the School Observation Measure (SOM) presentations prepare you for 
implementing the Formative Evaluation Process for School Improvement? 



APPENDIX D: 



Completed Checklist for Applying the Standards 
from The Program Evaluations Standards (1994) 



Checklist for Applying the Standards 



To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check as appropriate): 



Descriptor 

U 1 Stakeholder Identification 


The Standard was 
addressed 

X 


The Standard was 
partially addressed 


The Standard was 
not addressed 


The Standard was 
not applicable 


U2 Evaluator Credibility 


X 








U3 Information Scope and Selection 


X 








U4 Values Identification 


X 








U5 Report Clarity 


X 








U6 Report Timeliness and Dissemination 




X 






U7 Evaluation Impact 


X 








FI Practical Procedures 


X 








F2 Political Viability 


X 








F3 Cost Effectiveness 


X 








PI Service Orientation 


X 








P2 Formal Agreements 




X 






P3 Rights of Human Subjects 


X 








P4 Human Interactions 


X 








P5 Complete and Fair Assessment 


X 


* ^ 






P6 Disclosure of Findings 


X 








P7 Conflict of Interest 


X 








P8 Fiscal Responsibility 


X 








A1 Program Documentation 


X 








A2 Context Analysis 


X 








A3 Described Purposes and Procedures 


X 








A4 Defensible Information Sources 


X 








A5 Valid Information 


X 








A6 Reliable Information 


X 








A7 Systematic Information 


X 








A8 Analysis of Quantitative Information 


X 








A9 Analysis of Qualitative Information 


X 








AlO Justified Conclusions 


X 








All Impartial Reporting 


X 








A12 Metaevaluation 


X 









The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 



request for evaluation plan/design/proposal 

evaluation plan/design/proposal 

evaluation contract 

^ evaluation report 

other: 



Name Paige Py rish 



Position or Title 



Research Assistan 



^ignature) 



Agency 
Address , 



AEL , Inc , 



P,0, Box 1348 Charleston, WV 25325-1348 



n.tP 3/3/00 



Relation to Document First author 



(e.g., author of document, evaluation team leader, external auditor, internal auditor) 
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